{
  "cells": [
    {
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "markdown",
      "source": "# INTRODUCTION\n\nThis kernel is meant to help solve one of this competition's initial chellenges - unification of all available datasets to be able to use all data for teaching our model\n\nHere we will not be exploring data all that mutch, nor we will be engineering any new features. Instead we will unite all the data and see what kind of score we can get from this initial information using a few models\n\nIn this notebook we will perform the following operations:\n\n- combine all available datasets to form complete applicants statistics \n- prepare data for modeling\n    - impute missing values\n    - scale data\n- run baseline model - Logistic Regression\n- run advanced model - XGBOOST\n- compare results\n- check our feature importance (as per our xgb model oppinion)\n\nI will try to provide as detailed explaination as possible. \n\n**Any comments and or critisism are highly welcome =)**"
    },
    {
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "collapsed": true,
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": true
      },
      "cell_type": "code",
      "source": "import numpy as np\nimport pandas as pd\nimport warnings\nfrom sklearn.preprocessing import MinMaxScaler, Imputer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\nimport xgboost as xgb\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7a103e40eb4e1d4820f4aa0fec3e6b9d97263f59"
      },
      "cell_type": "markdown",
      "source": "We can see that our data is rather big (hundreds of megabytes for some files). This may create problems when running heavy calculations, so below I have used some really cool formula I found to reduce the size of data being loaded. It basically iterates through all the data types within a file and substitutes data for a lightest possible format. I found it here : https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "574c0cdcc17c0f1d4cedf4ecf62fccf7d3331ea2"
      },
      "cell_type": "code",
      "source": "def reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n       \n        1. Iterate over every column\n        2. Determine if the column is numeric\n        3. Determine if the column can be represented by an integer\n        4. Find the min and the max value\n        5. Determine and apply the smallest datatype that can fit the range of values\n\n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df\n\n\ndef import_data(file):\n    \"\"\"create a dataframe and optimize its memory usage\"\"\"\n    df = pd.read_csv(file, parse_dates=True, keep_date_col=True)\n    df = reduce_mem_usage(df)\n    return df",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "aac178ea138a84a5ac7e514cb27d59d0ebc3d97a"
      },
      "cell_type": "markdown",
      "source": "Now we can load in our data with this new function we've just created"
    },
    {
      "metadata": {
        "_kg_hide-output": false,
        "trusted": true,
        "_uuid": "68878f508a9e053713a3174c0ce13b81ae8079e4",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train = import_data('../input/application_train.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d3a87f78102788d15adbb7ec43c6ebb95859c1cb"
      },
      "cell_type": "markdown",
      "source": "You can clearly see the data weight reduction while keeping all the information in place\n\nDecreased by 79%"
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "9c815fd27e8b29ce196458dfb14454b159f97c0f",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "test = import_data('../input/application_test.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "a636b9dcb7820667c99a7974afaaf92cf661e145",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "bureau = import_data('../input/bureau.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "6fd8e415abf4cc33c404129fb1cbf75c229b6778",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "bureau_balance = import_data('../input/bureau_balance.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "0e5d6f56dae3f25679d17f4a1743bff032dac32a",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "previous_application = import_data('../input/previous_application.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "08b0798adf7e78686470ec09f50c4463ce48764e",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "credit_card_balance = import_data('../input/credit_card_balance.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "a1b3e35939e3232a382e8712f19464770abfa33f",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "installments_payments = import_data('../input/installments_payments.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "84a414494884b91bcca77659cca0fca90cc9f744",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "POS_CASH_balance = import_data('../input/POS_CASH_balance.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4060f7449c24a746fb9ecd7fccefeb36a9feff52"
      },
      "cell_type": "markdown",
      "source": "# INITIAL EXPLORATION"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "864e32ddf057157bb475eaac462d27f8a48c156c",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train.TARGET.value_counts()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f7fda8d77ac0a59fadb4c63b79cb245b336271a0",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# we can see that our binary target variable is heavily disbalanced and is distributed at roughly 8%/92%\ntrain.TARGET.plot.hist(color = 'g')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "60c3e47a30af41e563a28245727af8c067379e54"
      },
      "cell_type": "markdown",
      "source": "What about missing values?"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "87e3e2eec1de2212e3f374799dd61e3afcea3936",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# total percentage of NAs in data\nsum(train.isna().sum())/(train.shape[0]*train.shape[1])*100",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "5ae19d1225ec8a23a826b081770d2a9713be1dd0",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# now let's have a look which columns have what percentage of missing data (within train dataset only for now)\n\ntrain_NA = (np.round((train.isna().sum()/len(train)),3)).sort_values(ascending = False)*100\ntrain_NA = train_NA.sort_values(ascending = False)\ntrain_NA.head(10)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "1d723b40984e15ec6522a69585ea64130afedf56"
      },
      "cell_type": "markdown",
      "source": "Now let's have a look what data types do we have in our train set"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "25027a50426110e3660a8c69f1c6874d2775b31d",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train.dtypes.value_counts()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "218ff0942e38d7b18120e91552fe2d4c0a941693",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# what about subcategories within categorical data\ntrain.select_dtypes('category').apply(pd.Series.nunique, axis = 0) # not so many",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3e2fa8fa34e3a44cf1d204515dd5ac2df260d9f5"
      },
      "cell_type": "markdown",
      "source": "We will not cover visualizations and further exploratory analysis. Later on we can see all that we need within modeling and feature imporatnce matrix.\n\nAfter all - data speaks for itself ,)"
    },
    {
      "metadata": {
        "_uuid": "b41172548a08ecc3322816114ded05f81b0cd1c8"
      },
      "cell_type": "markdown",
      "source": "# UNITE ALL DATASETS\n\nThis is kinda tricky as data has different format and is represented differently\n\nOur head dataset will be called 'data' which is just a merge of train and test\n\nOther pieces we will add after some processing"
    },
    {
      "metadata": {
        "_uuid": "bd5ef3d161c819d4d7e65843b4b3f1fce3a27b8c"
      },
      "cell_type": "markdown",
      "source": "**train + test**\n\nthis one is easy: both datasets have exactly the same format with only TARGET column being present in train set as the only difference"
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "862ba6269204e72fe5913442051243093f152100",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "data = train.append(test)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ce3ac06c48ee660d8f95c7016e10dd7ebe51dd11",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# now just in case, let's check if we've got it right\ndata.TARGET.isna().sum() # same as number of test rows",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "7d5aa85ec2d53a15bada567b2e31252b8d3dd392",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "sum(data.SK_ID_CURR[data.TARGET.isna()] == test.SK_ID_CURR) # all is good",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "67081faeb9acabed6984a0886dfbb72df40affe0",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "sum(data.SK_ID_CURR.isin(test.SK_ID_CURR)) == len(test) # nothing else to prove",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0eabb8e5d8995b114613d7783d8a9bc1e1392a60"
      },
      "cell_type": "markdown",
      "source": "** + bureau**\n\nBefore we merge data with bureau, we need to merge bureau dataframe with related information in bureau_balance file\n\nWhat is the exact problem here:\n1. bureau dataframe comes from the Credit Bureau authority and displays one row for each credit the client from train/test dataset has taken previously. It is matched by SK_ID_CURR with train/test and where in train/test the SK_ID_CURR do not duplicate (1 for 1 client whom we are trying to classify) in most cases bureau dataframe has multipe indicies of the same client as he/she had applied to multiple loans previously.\n2. in turn bureau_balance even more extends the previous credit information on a greater scale. It contains a separate row for each month of history of every previous credit reported to Credit Bureau (bureau dataframe) and is related to bureau df via SK_ID_BUREAU.\n\nSo the approach we are going to use is to calculate mean of each statistical column out of these both dataframes to include these mean values as features of our clients whom we are trying to classify. For example: mean days overdue for all credits that the client had previously taken.\n\nI have to say that this approach leaves out some information such as categorical columns in some cases. For example the client with SK_ID_CURR = 666 had 7 credits in bureau dataframe, and when we collapse all these credits (grouped by one ID) into one line to indicate mean values for these credits, we will not be able to show a CREDIT_ACTIVE column that has different categorical values as Closed or Active for different previous credits. So this leaves room for some interesting feature engineering here.\n\nSteps that we need to take:\n1. Collapse bureau_balance dataframe to mean values grouped by SK_ID_BUREAU\n2. Merge this with bureau dataframe\n3. Collapse bureau dataframe to mean values grouped by SK_ID_CURR\n4. Merge what we've got with our data df"
    },
    {
      "metadata": {
        "_uuid": "6a7c1afe925412651822bf8be374c4fc83f87bc7"
      },
      "cell_type": "markdown",
      "source": "Even though we've decided not to perform any feature engineering, one useful feature here is just asking for it. Let's calculate the total number of previous credits taken by each client and include this in our statistics. I believe that kind of information would be quite useful. So let's quickly do that before executing our program defined above"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4bddf3f53a261681cbb1e1061c19a9ad265169de",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "previous_loan_counts = bureau.groupby('SK_ID_CURR', as_index=False)['SK_ID_BUREAU'].count().rename(columns = {'SK_ID_BUREAU': 'PREVIOUS_LOANS_COUNT'})\nprevious_loan_counts.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "22ab5f3752c4b568a2ebce9e48c43505bb1736b4"
      },
      "cell_type": "code",
      "source": "data = data.merge(previous_loan_counts, on = 'SK_ID_CURR', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "dad35f2567ce363a5ae9999134770f3384d3f1c9"
      },
      "cell_type": "markdown",
      "source": "Now back to merging with all the bureau and bureau_balance information"
    },
    {
      "metadata": {
        "_uuid": "93bcecc6c12c76c9b90048d9956e2b58b38fbbff"
      },
      "cell_type": "markdown",
      "source": "*STEP 1 - collapse bureau_balance*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "5b6f5b4edc1f0e5cce5aaf0602fc3cdb86bb64b4"
      },
      "cell_type": "code",
      "source": "# first define the formula for grouping rows by ID and calculating mean values\ndef extract_mean(x):\n    y = x.groupby('SK_ID_BUREAU', as_index=False).mean().add_prefix('BUR_BAL_MEAN_') \n    return y",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "d46bbe50f1a1588c373e08296b5cd380f39e9476"
      },
      "cell_type": "code",
      "source": "# apply formula to create bureau_balance dataframe grouped by SK_ID_BUREAU with mean values of all numerical columns\nbureau_bal_mean = extract_mean(bureau_balance)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "80ef2d59c292d69efffb137eac16774449dec533",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "bureau_bal_mean.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "11e131e497518dc406612bd2a980ad94b1ff0f66"
      },
      "cell_type": "markdown",
      "source": "As you can see, this dataframe does not include the bureau_balance categorical column STATUS\n\nAlso note that our formula has changed the name of the SK_ID_BUREAU, we need to change it back in order to use it when merging with bureau df\n\nOne might argue that we didn't need to add this .add_prefix(...) to our formula above, but when working with larger datasets below it will prove itself useful"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "6b1f323343e9d8897af630549f54d1aae28fe147"
      },
      "cell_type": "code",
      "source": "bureau_bal_mean = bureau_bal_mean.rename(columns = {'BUR_BAL_MEAN_SK_ID_BUREAU' : 'SK_ID_BUREAU'})",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0fde54d369dcb8f596fe7363a6282977c34041cb"
      },
      "cell_type": "markdown",
      "source": "*STEP 2 - merge with bureau*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "b2013aeb6dd96519a6492d58474b37c2bbc39176"
      },
      "cell_type": "code",
      "source": "bureau = bureau.merge(bureau_bal_mean, on = 'SK_ID_BUREAU', how = 'left')\nbureau.drop('SK_ID_BUREAU', axis = 1, inplace = True) # we don't need this internal ID anymore",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4f4cc059c3e834962ba926d5c0c052e536e3f19c"
      },
      "cell_type": "markdown",
      "source": "*STEP 3 - collapse bureau*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "37931eb0875b97a1949d3b7cafa41b6b049f84af"
      },
      "cell_type": "code",
      "source": "def extract_mean(x):\n    y = x.groupby('SK_ID_CURR', as_index=False).mean().add_prefix('PREV_BUR_MEAN_') # note that we have changed the ID to group by and the prefix to add\n    return y",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "7fb85673ed3ec6a7abd8be1d189729b0c8ca0464"
      },
      "cell_type": "code",
      "source": "bureau_mean_values = extract_mean(bureau)\nbureau_mean_values = bureau_mean_values.rename(columns = {'PREV_BUR_MEAN_SK_ID_CURR' : 'SK_ID_CURR'})",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3b18e07c217d5a3cb6f918c92bcf0edcd0c317d3",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "bureau_mean_values.head(5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ae164b6d033bf97c7b36f3522321e6870a0aad0a"
      },
      "cell_type": "markdown",
      "source": "Looks good. There are a few missing values although which we will deal with later"
    },
    {
      "metadata": {
        "_uuid": "807ee1649b1ac064a06879000bc7d731b1a4adff"
      },
      "cell_type": "markdown",
      "source": "*STEP 4 - merge bureau with data*"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f97dd25f9aa86ddb3e4b51eea2dd316814599596",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "data.shape",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "6da34d99214a11e590a462acaf266e82a8035497"
      },
      "cell_type": "code",
      "source": "data = data.merge(bureau_mean_values, on = 'SK_ID_CURR', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "fbd599b3892fab59bcf9248741f5e5ae1d28e8fa",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "data.shape",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3f0788245ca97178f2a267681ca8a49c1060b87d"
      },
      "cell_type": "markdown",
      "source": "So here we've created 13 new features and added them to our train/test dataset called 'data'"
    },
    {
      "metadata": {
        "_uuid": "f642f65f59c5aff0e8129a36bffca545e7a7b746"
      },
      "cell_type": "markdown",
      "source": "**+ previous_application**\n\nQuick information on this block of data: surprisingly... previous_application reflects clients' previous applications for loans to Home Credit.\nAs before, previous_application unfolds in a load of statistics with three other dataframes:\n- POS_CASH_balance - monthly balance snapshots of previous Point Of Sale s and cash loans that the applicant had with Home Credit (one row for each month of history)\n- installments_payments - repayment history for previous credits with Home Credit (one row for each payment)\n- credit_card_balance - monthly balance snapshots of applicant's credit cards (one row for each month of history)\n\nThese 4 datasets have their own key to for internal mapping - SK_ID_PREV\n\nSo the plan here would be the following:\n\n1. Collapse credit_card_balance dataframe to mean values grouped by SK_ID_PREV\n2. Merge with previous_application (our 'leading' dataset in this case)\n3. Collapse POS_CASH_balance to mean values grouped by SK_ID_PREV\n4. Merge with previous_application\n5. Collapse installments_payments ...\n6. Merge with previous_application\n7. Collapse the resulting previous_application dataset to mean values grouped by SK_ID_CURR\n8. Merge our unfolded previous_application statistics with our data"
    },
    {
      "metadata": {
        "_uuid": "ed9839c35ae3bfb6c491782b9250ba5fba09c23e"
      },
      "cell_type": "markdown",
      "source": "But before we start, let's check if there are any records in previous_application that are not in our data?"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "2f473ab3acb2d07cb8cf6a6347e623c93fff5478",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "len(previous_application.SK_ID_CURR.isin(data.SK_ID_CURR)) == len(previous_application)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "04f6717e2dbaf209fd63ac6ad43bfd622a10a647"
      },
      "cell_type": "markdown",
      "source": "looks good"
    },
    {
      "metadata": {
        "_uuid": "5fc2d689722439f37b6cd70d69891eafd227bdc7"
      },
      "cell_type": "markdown",
      "source": "One more thing! We will delete the SK_ID_CURR from the credit_card_balance / POS_CASH_balance / installment_payments as we do not need this column to be shown as mean, this information has no impact on statistics and will just clutter the space as noise. We will group them with our 'leading' dataset previous_application using SK_ID_PREV and our 'leading' dataset has this SK_ID_CURR key to be further mapped with our data"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "f71a8d0686760b91b263423e2d92d81184207859"
      },
      "cell_type": "code",
      "source": "credit_card_balance.drop('SK_ID_CURR', axis = 1, inplace = True)\ninstallments_payments.drop('SK_ID_CURR', axis = 1, inplace = True)\nPOS_CASH_balance.drop('SK_ID_CURR', axis = 1, inplace = True)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "486b8b7874f246ae29316b7016d7cfefc99ff26d"
      },
      "cell_type": "markdown",
      "source": "As previously, before tearing apart the previous_applications to Home Credit statistics, let's extract the number of previous applications of the clients to Home Credit and add this feature to our data"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "581212d7e0518a869c6c7c8d9bc33849ed366ec4",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "previous_application_counts = previous_application.groupby('SK_ID_CURR', as_index=False)['SK_ID_PREV'].count().rename(columns = {'SK_ID_PREV': 'PREVIOUS_APPLICATION_COUNT'})\nprevious_application_counts.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "398ca5c35e04065c30a68d2dde2678086c72d477",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# and throw that column in our data\ndata = data.merge(previous_application_counts, on = 'SK_ID_CURR', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "8de691f6fc6d7e98f8748a3d6e53ee0e7642258d",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "data.head(5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ebea997d4f65a1184b01eb6373d89fe0ae5cd00a"
      },
      "cell_type": "markdown",
      "source": "Now back to our process"
    },
    {
      "metadata": {
        "_uuid": "e386107406e3b4ac764f47bc32828f7d4a0a0966"
      },
      "cell_type": "markdown",
      "source": "*STEP 1 - collapse credit_card_balance*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "9668c58ef06add96e47f6149ba26127ade3f1325"
      },
      "cell_type": "code",
      "source": "def extract_mean(x):\n    y = x.groupby('SK_ID_PREV', as_index=False).mean().add_prefix('CARD_MEAN_')\n    return y\n\ncredit_card_balance_mean = extract_mean(credit_card_balance)\n\ncredit_card_balance_mean = credit_card_balance_mean.rename(columns = {'CARD_MEAN_SK_ID_PREV' : 'SK_ID_PREV'})",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a997d4ed5acc14f2cc1684c090eb8b3fbf67d2b3"
      },
      "cell_type": "markdown",
      "source": "*STEP 2 - merge with previous_application*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "ecd04a8b5e010c56eeab0c4c99c719ec7aed44b3"
      },
      "cell_type": "code",
      "source": "previous_application = previous_application.merge(credit_card_balance_mean, on = 'SK_ID_PREV', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d290539fe993ea9c0e939dde0b986ba88b8d2392"
      },
      "cell_type": "markdown",
      "source": "*STEP 3 - collapse installments_payments*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "9c61e0011c6fffddf0329eaf2bbbc382b29d6aeb"
      },
      "cell_type": "code",
      "source": "def extract_mean(x):\n    y = x.groupby('SK_ID_PREV', as_index=False).mean().add_prefix('INSTALL_MEAN_')\n    return y\n\ninstall_pay_mean = extract_mean(installments_payments)\n\ninstall_pay_mean = install_pay_mean.rename(columns = {'INSTALL_MEAN_SK_ID_PREV' : 'SK_ID_PREV'})",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "fbc54677f40762c35dd6e7dbdbbcf9af00bf0a1a"
      },
      "cell_type": "markdown",
      "source": "*STEP 4 - merge with previous application*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "6a00b0741345a9b0afbeaa96754f54975baa34b7"
      },
      "cell_type": "code",
      "source": "previous_application = previous_application.merge(install_pay_mean, on = 'SK_ID_PREV', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": false,
        "_uuid": "05f5b03c220e3fa760d6b61fdaa73f17c52ab935"
      },
      "cell_type": "markdown",
      "source": "*STEP 5 - collapse POS_CASH_balance*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "0db6fa1b26677d85f3d9cc5a3cd6d41b8eeba2b2"
      },
      "cell_type": "code",
      "source": "def extract_mean(x):\n    y = x.groupby('SK_ID_PREV', as_index=False).mean().add_prefix('POS_MEAN_')\n    return y\n\nPOS_mean = extract_mean(POS_CASH_balance)\n\nPOS_mean = POS_mean.rename(columns = {'POS_MEAN_SK_ID_PREV' : 'SK_ID_PREV'})",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "835a80cd3ce9a51ab8d1de9bc6cc4b0443368838"
      },
      "cell_type": "markdown",
      "source": "*STEP 6 - merge with previous_application*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "b49d2a5e4c7518d20e4dfec4d4651e32b3bee858"
      },
      "cell_type": "code",
      "source": "previous_application = previous_application.merge(POS_mean, on = 'SK_ID_PREV', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b6522578402e7c53999dd56b35359d876214216f"
      },
      "cell_type": "markdown",
      "source": "*STEP 7 - collapse the resulting previous_application dataset to show mean values grouped by SK_ID_CURR*"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "24d900ed383cbde31dce0651e72d7100277d256c"
      },
      "cell_type": "code",
      "source": "def extract_mean(x):\n    y = x.groupby('SK_ID_CURR', as_index=False).mean().add_prefix('PREV_APPL_MEAN_')\n    return y\n\nprev_appl_mean = extract_mean(previous_application)\n\nprev_appl_mean = prev_appl_mean.rename(columns = {'PREV_APPL_MEAN_SK_ID_CURR' : 'SK_ID_CURR'})\n\nprev_appl_mean = prev_appl_mean.drop('PREV_APPL_MEAN_SK_ID_PREV', axis = 1) # we don't need this intermediate column any more",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "73a3a562b661a0401b27f966eed232d49cf21f44"
      },
      "cell_type": "markdown",
      "source": "*STEP 7 - merge what we've got with our data*"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ac147a3719947356cb10aab5a16c7248ad790d97",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "print('data shape', data.shape)\nprint('previous applications statistics shape', prev_appl_mean.shape)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "ce7306896ca9bc73d4f0deffe9678a295ca7d4f0"
      },
      "cell_type": "code",
      "source": "data = data.merge(prev_appl_mean, on = 'SK_ID_CURR', how = 'left')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7070b518a897838700e16b9a42998f8d0f53ba75"
      },
      "cell_type": "markdown",
      "source": "As we can see, this last sprint over previous applications added 50 new features to our statistics and completed the unification of all data"
    },
    {
      "metadata": {
        "_uuid": "022f10f8bdebc0cdc1792e33e0bad4e4b8f2fc27"
      },
      "cell_type": "markdown",
      "source": "# PREPROCESS DATA\n\nUp next:\n1. Split for train and test datasets\n2. One-hot encode categorical columns\n3. Handle missing values\n4. Scale data"
    },
    {
      "metadata": {
        "_uuid": "f6199b65c93fd6cf70609dc5a085d7d564662aa4"
      },
      "cell_type": "markdown",
      "source": "**Split**\n\nPerform split according to IDs in initial train and test datasets"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c3866ee64b5b1338f43017abce8971f0a8de5ca5",
        "_kg_hide-output": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train1 = data[data['SK_ID_CURR'].isin(train.SK_ID_CURR)]\n\ntest1 = data[data.SK_ID_CURR.isin(test.SK_ID_CURR)]\ntest1.drop('TARGET', axis = 1, inplace = True)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ae5cd2ce7e5d911ee976b9ecdf2f59492ca143af",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "print('Training Features shape with categorical columns: ', train1.shape)\nprint('Testing Features shape with categorical columns: ', test1.shape)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": false,
        "_uuid": "da9f21116e35055ef47bed6bbf4bebd2ba67c45a"
      },
      "cell_type": "markdown",
      "source": "**One-hot encode or create dummy variables**\n\nWhat that is - is the representation of categorical columns that have two or more categories inside them as their own columns with binary numeric representation (0 or 1). For example in the initial dataframe you've got an EDUCATION variable which has subcategories as Primary/Middle/Higher/Phd/etc. Machine learning algorithms can not handle such data. So instead of storing all this information in one column called EDUCATION, we create 4 new separate columns according to subcategories and as a result for one client with Phd degree we will have 4 columns Primary / Middle / Higher / Phd. First three will display 0, as the applicant does not have Primary / Middle / Higher, and will display 1 in the Phd column. Thus our categorical column became numeric."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4e1ecee3c719d2186793657b338d0ee60bcde09e",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train1 = pd.get_dummies(train1)\ntest1 = pd.get_dummies(test1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "8659bce8dfd6ee6dfbf190a4529020cb0b6b9c96",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# After one-hot encoding it is usually a good idea to check the shapes again\n\nprint('Training Features shape with dummy variables: ', train1.shape)\nprint('Testing Features shape with dummy variables: ', test1.shape)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "894b9e22493d7cf1f53eea338a1fdb472bcfbe1e"
      },
      "cell_type": "markdown",
      "source": "Looks like training data had more subcategories as the number of columns higher that in test set by 4. Actualy by 3, because we still have target in our train1 dataset\n\nWe can not have different features that categorize data in the training and testing datasets, so let's make them even"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "2adfb95a1f988a6e61f61507e06f0cd45bcc7df7"
      },
      "cell_type": "code",
      "source": "TARGET = train1.TARGET # save our TARGET variable\ntrain1.drop('TARGET', axis = 1, inplace = True) # remove TARGET from train1\n\n# allign the datasets\ntrain1, test1 = train1.align(test1, join = 'inner', axis = 1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1acca728dcef56cbe524e3bad78bdaef9a3097f9",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "print(train1.shape)\nprint(test1.shape)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "84b71f0761b872c5f4619ad09a746a711a5b47dd"
      },
      "cell_type": "markdown",
      "source": "**MISSING VALUES**\n\nAs a baseline model I've chosen logistic regression. For this algorithm in particular, and in general we need to deal with missing values. \n\nThere are various methods such as pridicting the NAs based on other statistics, but we will use one of the simpler approaches and just fill in the mean or median values for the corresponding columns"
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "31fcc3f1ac664f2a37bdf8cb4d8cfd93d3e9ac94",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Define median imputation of missing values\nimputer = Imputer(strategy = 'median')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bde479d543eb43573d6e153b8008d914a3851afd",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "print('Missing values in train data: ', sum(train1.isnull().sum()))\nprint('Missing values in test data: ', sum(test1.isnull().sum()))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b917e4422d3ff64fb88d19c2c49635cba1133c56",
        "_kg_hide-output": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Fit imputer to our train dataset\nimputer.fit(train1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "164227d12a4544eed6a2506c86a461d3fbecef0e",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Apply imputer to our dataframes\nimputed_train = imputer.transform(train1)\nimputed_test = imputer.transform(test1)\n\n# the reason we created new objects (imputed_train/imputed_test) instead of just overriding train/test is because imputer creates \n    # arrays that include all the non missing values and imputed missing values, but it completely discards \n        # features names, which we will need later to examine feature importance",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9e1215f658129c60fb43a05d1a56d33573d59f16",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# now before we replace missing values by imputed, let's make sure we understand what will happen\ntrain1.head(5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "22a1e003e5a133e64812facb949e0e3d16952041"
      },
      "cell_type": "markdown",
      "source": "Scroll right to APARTMENTS_AVG column to see the NaNs"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b2e238fcf4ac1dbd9e05c4d4374a3ce98e7088ec",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# manually check what is the median of this column\ntrain.APARTMENTS_AVG.median()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "eedc21d8e95b595dd4846fd6d44d9524aeffc687"
      },
      "cell_type": "code",
      "source": "# replace the missing values with imputed ones\ntrain1[train1.isnull()] = imputed_train\ntest1[test1.isnull()] = imputed_test",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "5e9f44c6fddf1c4873c0b16657b67cd5deba5e39"
      },
      "cell_type": "markdown",
      "source": "look again at this APARTMENTS_AVG column where those NaNs were and you can see that those missing values got replaced with median for this column that we had checked previously 0.087585"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bd753a0fee7949ce5edf685fd3ed24b104df596c",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train1.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0f7ca0601cc1dbd73c81f731e33b35e970ded187",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "sum(train1.isnull().sum()) # we've gotten rid of all our NAs",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "976ddf198ee6edffc90464ee217d048d7651a134"
      },
      "cell_type": "markdown",
      "source": "**Scaling the data**\n\nWhat is scaling? \nFor example one person has a salary 100K, the other one, not as lucky, has 10K. Other parameter for example number of kids varies from 0 to 5. These two factors are in totally different orders: one being in hundreds of thousands other being in units. There are different scaling methods. The one I chose represents each column in values from 0 to 1 according to their relative values.\n\nWhat is it for?\nBasically, it creates less variance in data and smoothes it out to relative values making them easier to compare to each other."
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "2207a09b4baeed4008984548ba29df5a97d28015"
      },
      "cell_type": "code",
      "source": "# Define what type of scaling we will use \nscaler = MinMaxScaler(feature_range = (0, 1))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "595900d1798104391568098adf7e8be0370a0578",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Fit scaler to our training data\nscaler.fit(train1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "28ad6df5461dcba63955244dbd7eae28f7cb5902"
      },
      "cell_type": "code",
      "source": "# Calculate scaled values and store them in a separate object\nscaled_train = scaler.transform(train1)\nscaled_test = scaler.transform(test1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0ee36541a3263ae10c397091d94f9c4fc2ca2319",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# As before, in order to keep our column names we include scaled_values to our train/test dataframes like this\n\ntrain1 = pd.DataFrame(scaled_train, index=train1.index, columns=train1.columns)\ntest1 = pd.DataFrame(scaled_test, index=test1.index, columns=test1.columns)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "76a639d16960d27714dbbb5cab24e220874fe2d4",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train1.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d1a549a168130f22f7dd3ab3bebfcc99322fbb3f"
      },
      "cell_type": "markdown",
      "source": "# MODELING\n\nBasically we now have a dataset that includes all the features (not counting the droped categorical columns in bureau and previous_application data) all imputed and preprocessed\n\nWe can now try a few models to see the baseline result and compare it to some more advanced models as well as observe performance shifts introducing new features\n\n**Baseline model - Logistic Regression**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d50d418351c4f9efbfe4aa917b0cf8c1387aa076",
        "_kg_hide-output": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Define our model\nGLM = LogisticRegression(C = 0.0001)\n\n# Fit the model to the training data\nGLM.fit(train1, TARGET)\n\n# Predict on the testing data\nGLM_pred = GLM.predict_proba(test1)[:, 1] # use predict_proba to predict probabilities and \n                                            # [:,1] for the 1 probability output (by default it will save both probabilities, \n                                                # for value 0 and value 1)\n# Create the submission file\nsubmission = test[['SK_ID_CURR']]\nsubmission['TARGET'] = GLM_pred",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "2acf7ccafdd13546031398941a11b2faf0b85ce2",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "submission.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "205953b8dfd67bef65c2c1042b63e1f28aee228e"
      },
      "cell_type": "code",
      "source": "# Save our submission to output\nsubmission.to_csv('GLM_baseline.csv', index = False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "5a6595342b8546ea3d0b9e55e651a8f279400418"
      },
      "cell_type": "markdown",
      "source": "Result: \n- AUC 0.682\n- Liederboard position 2100ish\n\nIf not for liederbord comparison, in real life we could not really tell wether this is good or bad\n\nWe can treat is a baseline that we should try to improve"
    },
    {
      "metadata": {
        "_uuid": "59177aa3d5db878088c48a823184193c58192475"
      },
      "cell_type": "markdown",
      "source": "**Advanced model - XGBOOST**\n\nAs an experiment we will run a more advanced model on the same data to see what could be the difference in prediction quality"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "568f373018627d51b403f321c1e421db6c31ab9f",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Give appropriate names to our key objects \n# (doesn't make any difference, but those are the correct names to be used not to get confused)\n\nX_train = train1\ny_train = TARGET\n\nX_test = test1\n\n# Create DMatrix objects (necessary for xgboost model)\ndtrain = xgb.DMatrix(X_train, y_train)\ndtest = xgb.DMatrix(X_test)\n\n# Define model parameters\n# These parameters are almost random, all of them really really should be \n    # tuned within cross validation, but that lies beyond our scope\n\nparams = {\n    'min_child_weight': 19.0,         # minimum number of finally classified records in one leaf (the classifier will not break leaf further into pieces when the leaf size has reached 19 instances)\n    'objective': 'binary:logistic',   # our objective is binary classificaiton\n    'max_depth': 7,                   # maximum tree depth\n    'eta': 0.025,                     # learning rate\n    'eval_metric': 'auc',             # it is very important to train the model with a specific metric in mind, in our case - AUC\n    'max_delta_step': 1.8,              # correct metric for train/valid is 20% success, because you can be doing a great job preparing data\n    'colsample_bytree': 0.4,               # engineering features, etc, and teach the model to perform good at incorrect metric...\n    'subsample': 0.8,\n    'gamma': 0.65\n    }\n\n# This will verbose model training progress\nwatchlist = [(dtrain, 'train')]\n\nXGB_model = xgb.train(params, dtrain, \n                300,                  # numrows\n                watchlist,            # to see training performance (not really relevant without validaiton set)\n                verbose_eval=50)      # to show training progress every 50 rounds\n\n# Predict on the test data\nXGB_pred = XGB_model.predict(dtest)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "50a0c3ae65878328853b6371067d69743f2bdd0a",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# Create submission file\nsubmission_2 = test[['SK_ID_CURR']]\nsubmission_2['TARGET'] = XGB_pred",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "894a7a2dfed2c27ab8e38c1ff124a51d16019902",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "submission_2.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "6b4f27974ddd8d7353cef813da06cbe9223380c4"
      },
      "cell_type": "code",
      "source": "# Export sumbmission\nsubmission_2.to_csv('XGB_pred2.csv', index = False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8b24670b84b15a78ea021b0d658fc041b2112374"
      },
      "cell_type": "markdown",
      "source": "Result: \n- AUC 0.763\n- Liederboard position 1500ish\n\nA pretty impressive jump over roughly 600 positions up the ladder with just changing the model type\n\nAlthogh the result is far from perfect, this experiment demonstrated that xgboost is referred as \"kaggle competition winning algorithm\" for a reason"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ecc13bb5413c01b40d512b9c5ded053f3428b19f",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "sns.set(font_scale = 1.2)\nfig, ax = plt.subplots(figsize=(20, 40))\n\nxgb.plot_importance(XGB_model, ax = ax)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "007d4eacd04e31151694ec3dfb0f12cb409b6a70"
      },
      "cell_type": "markdown",
      "source": "Oups... I've forgotten that we've got 300+ features, which makes this plot cluttered no matter how large we make it\n\nLet's extract importances values and construct our own plot with the top variables"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b1c61e217bb3da6239b2d3b69d329882ba5e6bef",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "importances = XGB_model.get_score()\n\nimportances = pd.DataFrame({'Importance': list(importances.values()), 'Feature': list(importances.keys())})\nimportances.sort_values(by = 'Importance', ascending = True,  inplace = True)\nimportances[importances['Importance'] > 200].plot(kind = 'barh', x = 'Feature', figsize = (8,12), color = 'orange')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8324714ba81ff349f71126d0fa6ce4cd0a230680"
      },
      "cell_type": "markdown",
      "source": "This is a good alternative to initial data exploration. It gives us an unbias picture about which variables are actually important to the resulting prediction"
    },
    {
      "metadata": {
        "_uuid": "94ec753a522f11783a2f1fd0c1e833685e40ce59"
      },
      "cell_type": "markdown",
      "source": "# CONCLUSION\n\nI believe that we have fulfilled the purpose of this notebook:\n\n1. Combined all the given data\n2. Performed basic minimum preprocessing\n3. Created baseline model\n4. Created advanced model to check the improvement potential\n5. Created great basis for further improvement\n\nWhat's next:\n-\tInclusion of the (dropped during unification) categorical data (in numeric format)\n-\tFeature engineering. I can not stress this more than saying that extraction of new features is crucial for metrics improvement\n-   Change missing values imputation strategy\n-\tModel tuning\n    - Cross validation\n    - Model ensembling\n\nYour imagination is the limit…"
    },
    {
      "metadata": {
        "_uuid": "6c29fab3297a983860bcfb439e34783301cb82dd"
      },
      "cell_type": "markdown",
      "source": ""
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}